Computer system backup performance optimization through performance analytics

ABSTRACT

Embodiments in accordance with the present invention disclose a method, computer program product, and system for optimizing performance of a computer backup solution that includes at least two data movers. The automated method includes measuring data mover performance during operation of a backup cycle, and optimizing the performance of data movers by increasing or decreasing the number of threads operating concurrently in the data movers. The method further includes computation of performance rankings of the data movers and shifting workload among the data movers in accordance with their respective performance rankings, such that the computer backup solution converges toward an optimized configuration.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of computer systembackup, and more particularly to dynamically optimizing performance of acomputer system backup solution.

In information technology, a “backup” refers to the copying andarchiving of computer data so it may be used to restore the originaldata. Backed up data may be stored on one or more servers, which may begeographically remote from the source system and each other. Backupshave at least two distinct purposes, one of which is to enable recoveryof data after it has been lost, for example due to deletion, hardwarefailure or physical loss, a disastrous event, etc. Another purpose ofbackups may be to recover a previous version of data following anerroneous or premature update.

Performing backups from large computer systems presents challenges inmanaging and moving large volumes of data from source machines to thebackup servers in a time efficient, and cost efficient manner.

SUMMARY

Embodiments in accordance with the present invention disclose a method,computer program product, and system for optimizing performance of acomputer backup solution for virtual machines that includes at least twodata movers. Data movers are the logical processes that transfer backupdata from the source virtual machine to the backup server. The automatedmethod includes measuring data mover performance during operation of abackup cycle and optimizing the performance of data movers by increasingor decreasing the number of threads operating concurrently in the datamovers. The method further includes computation of performance rankingsof the data movers and shifting workload among the data movers inaccordance with their respective performance rankings, such that thecomputer backup solution converges toward an optimized configuration.

According to an aspect of the present invention, there is a method,computer program product and/or system for optimizing performance of acomputer backup solution that includes at least a first data mover and asecond data mover, that performs the following steps (not necessarily inthe following order): (i) measuring performance statistics of the firstdata mover with respect to at least one virtual machine class processedby the first data mover, to produce performance statistics, by virtualmachine class, of the first data mover; (ii) measuring performancestatistics of the second data mover with respect to at least one virtualmachine class processed by the second data mover, to produce performancestatistics, by virtual machine class, of the second data mover; (iii)optimizing, by one or more processors, performance of a data mover; (iv)operating, by one or more processors, a performance model operationallycoupled with a data mover wherein the performance model includes atleast performance statistics of the first data mover and performancestatistics of the second data mover; (v) initiating, by one or moreprocessors, communications between the first data mover and the seconddata mover during at least a backup cycle, to exchange at leastperformance statistics between the first data mover and the second datamover; (vi) generating, by one or more processors, a peer-to-peer modelwherein the first data mover has access to the performance model of thesecond data mover and the second data mover has access to theperformance model of the first data mover; (vii) computing, by one ormore processors, performance rankings with respect to virtual machineclass, of a first data mover and a second data mover, based at least inpart on performance statistics of the first data mover and performancestatistics of the second data mover; (viii) analyzing, by one or moreprocessors, respective performance statistics of the first data moverand the second data mover, with respect to virtual machine class, toproduce performance rankings, by virtual machine class, of the firstdata mover and the second data mover; (ix) shifting, by one or moreprocessors, some workload from a first data mover to a second datamover, in accordance with their respective performance rankings, withrespect to virtual machine class, such that the computer backup solutionconverges toward an optimized configuration; (x) shifting, by one ormore processors, some workload from virtual machines of a class, fromthe first data mover to the second data mover, based at least in part,on the performance rankings of the first data mover and the second datamover with respect to workload from virtual machines of the class; (xi)operating concurrently, one or more producer threads in the data mover;(xii) operating concurrently, one or more consumer threads in the datamover; (xiii) changing the number of producer threads or consumerthreads operating concurrently in the data mover; (xiv) measuring achange of performance of the data mover, in response to changing thenumber of producer threads or consumer threads operating concurrently inthe data mover, to produce performance statistics on which to base, atleast in part, a subsequent change in the number of producer threads orconsumer threads operating concurrently in the data mover, such that thedata mover converges toward an optimized performance state; (xv) storingat least performance statistics relative to a first data mover, in theperformance model associated with the first data mover; (xvi) storing atleast performance statistics relative to a first data mover, in theperformance model associated with the second data mover; (xvii) storingat least performance statistics relative to a second data mover, in theperformance model associated with the first data mover; and (xviii)storing at least performance statistics relative to a second data mover,in the performance model associated with the second data mover.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a generic representation of a computing environment, generallydesignated with numeral 100, within which embodiments in accordance withthe present invention may operate.

FIG. 2 represents a data mover 115, generally designated with numeral200, in an embodiment in accordance with the present invention.

FIG. 3A is a flowchart depicting the top level functions of backupsolution 105 operating in an embodiment in accordance with the presentinvention, and generally designated with numeral 300A.

FIG. 3B, is a flowchart, generally designated with numeral 300B,illustrating functions of high level thread 205, performed by a localdata mover 115 in at least one embodiment in accordance with the presentinvention.

FIG. 3C is a flowchart illustrating functions of a low level thread 210,generally designated with numeral 300C, performed by data mover 115,operating in at least one embodiment in accordance with the presentinvention.

FIG. 4 depicts a block diagram of components of data processing system400, representative of any computing system within data processingenvironment 100 in accordance with an illustrative embodiment of thepresent invention.

DETAILED DESCRIPTION

The architecture of a virtual machine backup solution is often hamperedby an inability to effectively place, or efficiently optimize, theallocation of workload among data movers processing virtual machine datafor ingestion into a backup product. For example, data movers may beplaced on host systems but non-optimally positioned such that they failto provide the best aggregate throughput for their assigned virtualmachine workload. Furthermore, a set of data movers may have the overallworkload sub-optimally partitioned among them.

In general, a virtual machine backup solution may be deployed manually,by an individual or a team. Available planning assistance may include“best practices” guides, worksheets, or simple calculators intended tohelp a user to estimate environmental resource requirements and togenerate hints about where to place data movers for optimal efficiency.Often missing is the ability to validate environmental assumptions andreact dynamically, automatically, and analytically to actual performanceonce a backup solution has been deployed. System administrators,attempting to seek improvements, often perform optimization manually,through an iterative trial and error process, as backup systemperformance is evaluated and modifications made.

Another drawback of existing virtual machine backup solutions is theirinability dynamically to re-allocate the workload across data moversand, for a given data mover, during and between backup cycles.Configuration settings govern the workload allocated to a given datamover, and aspects of how that workload is processed by the data mover.Updates to the configuration settings are generally made by manualinput.

A backup solution is an overall backup system that, among other things,governs the operation of data movers, and determines an optimal mappingof data movers to virtual machine classes. A data mover is a clientbackup application instance which reads data from a virtual machineinfrastructure and sends the data to a backup server. A data mover canbe a physical data mover, or a virtual data mover. A physical data moverresides on a physical host system whereas a virtual data mover is a datamover residing on a virtual machine.

A backup cycle is one complete backup operation, wherein data from thesystem being backed up is copied from the system to the backup storage.Typically, a backup cycle occurs within a pre-defined time intervalreferred to as a backup window. A backup cycle can be triggeredautomatically, for example to occur on a pre-defined schedule, or inresponse to user input or other events.

Embodiments in accordance with the present invention implement alogic-based computer system backup solution, to enable automaticmodification of the backup solution configuration. Modifications of thebackup solution configuration can take place during backup cycles, andbetween backup cycles, to optimize performance of the backup solution.

A plurality of data movers use a peer-to-peer model to communicate witheach other during operation of a backup cycle. In cases where two datamovers need to communicate and there is no direct network connectionbetween them, the peer-to-peer model includes an ability to route thecommunications through one or more servers. During a backup cycle,various performance statistics are monitored and recorded to provide abasis on which to modify configuration of the backup solution. During abackup cycle, performance of each data mover is continually monitored.Aspects of the backup solution configuration can be modifieddynamically, for example, by shifting portions of the total backupworkload among the data movers, such that the shifted portions of theworkload are migrated to data movers proving to have higher throughputthan the throughput achieved by the data movers away from which theworkload is shifted.

The term “throughput” should be understood by one skilled in therelevant art to be a measure of an aggregate rate of data transfer. Thethroughput may be measured in megabytes per second, or other suitableunits of data transfer.

The terms “local” and “peer” are used herein with reference to datamovers. Any data mover in a backup solution can be considered to be alocal data mover. When discussion is focused on a particular data mover,the particular data mover is referred to as the local data mover andother data movers in the backup solution are considered to be peer datamovers.

If performance analysis determines that a local data mover is fasterthan one or more peer data movers, at processing workload from virtualmachines of a first class, and slower than one or more peer data movers,at processing workload from virtual machines of a second class, thelocal data mover may subsequently be assigned a larger proportion of itsworkload from virtual machines of the first class, and assigned asmaller proportion of its workload from virtual machines of the secondclass. Workload reassignment can occur dynamically, during operation ofa backup cycle, or between backup cycles, wherein the reassignedworkload can be implemented in a subsequent backup cycle.

In some embodiments in accordance with the present invention, virtualmachines are classified according to certain characteristics, such asfor example, its hostname, its data store location, its network path orother characteristics. Taken together, the characteristics areconsidered to define a particular virtual machine class. Any number ofrelevant characteristics can be chosen to classify virtual machines in away intended to be meaningful with regard to backup performance. If aparticular classification scheme proves to be ineffective in theoptimization process, other classification schemes can be implemented.

Furthermore, performance statistics from the data movers, as well asfrom the overall backup solution, are analyzed automatically, during andbetween backup cycles. Based on the analysis results, mapping of datamovers to various classes of virtual machine can automatically bereconfigured, thus continually optimizing overall performance of theindividual data movers leading to optimization of the overall backupsolution. Optimization is thus automatic, dynamic and responsive tochanges as the computer system, and its configuration and operatingparameters, evolve during backup cycles and over time.

Performance statistics can comprise various types of data pertaining tothe performance of a data mover, to provide a basis on which meaningfulperformance comparisons between a plurality of data movers can bederived. The particular types of data comprising performance statisticsgathered in some embodiments in accordance with the present inventionare implementation aspects.

In embodiments in accordance with the present invention, the backupsolution employs a peer-to-peer model wherein, either directly orthrough one or more intermediate servers, the plurality of data moverswithin the backup solution automatically communicate among themselves. Adata mover is a peer to each other of the plurality of data moversincluded in the backup solution. Data movers exchange performancestatistics in the peer-to-peer model to continually self-optimize. Moredetailed information with regard to the peer-to-peer model can be foundwith reference to FIGS. 3A through 3C and the relevant discussionsthereof.

An internal performance model is maintained and updated by each datamover, and communicated among its peer data movers. In some embodimentsin accordance with the present invention, performance models from thedata movers are also communicated to the backup solution as well. Aperformance model comprises at least normalized “performance ratios” forthe virtual machine classes processed by a data mover.

In some embodiments in accordance with the present invention, thefollowing formula is used to compute a normalized performance ratio fora data mover:

${PR}_{1} = \frac{{Throughtput}\mspace{14mu}{of}\mspace{14mu}{DM}_{1}}{{Combined}\mspace{14mu}{throughput}\mspace{14mu}{of}\mspace{14mu}{all}\mspace{14mu}{DMs}}$

Where:

PR₁=Normalized performance ratio of data mover DM₁ operating within thebackup solution, and DM₁ processing data from virtual machines of aparticular class.

Throughput of DM₁=Throughput of data mover DM₁, operating within thebackup solution, and DM₁ processing data from virtual machines of theparticular class.

Combined throughput of all DMs=Total throughput of all data movers,including DM₁, operating within the backup solution, the data moversprocessing data from virtual machines of the particular class.

Example computations of normalized performance ratios follow:Hypothetical performance figures for data movers DM₁ and DM₂, both datamovers processing data from virtual machines of classes “A” and “B”(VM_(A) and VM_(B) respectively) are given. In this example, thefollowing throughput results (expressed in MB/sec or other suitable unitof measure) given as:

DM₁ processes VM_(A) at 40, and VM_(B) at 55.

DM₂ processes VM_(A) at 30, and VM_(B) at 60.

The normalized performance ratios (unitless) of DM₁ and DM₂ can becomputed as follows:

Normalized Performance Ratios DM₁ DM₂ VM_(A) $\frac{40}{40 + 30} = 0.57$$\frac{30}{40 + 30} = 0.43$ VM_(B) $\frac{55}{55 + 60} = 0.48$$\frac{60}{55 + 60} = 0.52$

From the perspective of the data movers, the results indicate that DM₁has a higher performance ranking when processing VM_(A) than does VM_(B)(0.57 vs. 0.48 respectively), and DM₂ has a higher performance rankingwhen processing VM_(B) than does VM_(A) (0.52 vs. 0.43 respectively).Accordingly, from the perspective of the virtual machines, it can beseen that VM_(A) is processed faster by DM₁ than by DM₂ (0.57 vs. 0.43respectively) whereas VM_(B) is processed faster by DM₂ than by DM₁(0.52 vs. 0.48 respectively). Based at least on these rankings,embodiments in accordance with the present invention would tend to shiftat least some workload from virtual machines of class “A” from DM₂ toDM₁ and to shift at least some workload from virtual machines of class“B” from DM₁ to DM₂, thus approaching an optimized configuration for thebackup solution. Some or all of the workload shifting may occur duringoperation of the backup cycle or at initialization or operation of asubsequent backup cycle.

To further optimize performance of a data mover, embodiments inaccordance with the present invention utilize multiple parallelsessions, referred to herein as “threads,” operating concurrently in thedata mover. During a backup cycle, a data mover can dynamically increaseor decrease the number of threads it uses. A data mover makes use of aplurality of “producer” threads, wherein data is read into the datamover from a virtual machine. Moreover, a data mover makes use of aplurality of “consumer” threads, wherein data is forwarded from the datamover to the backup server. The plurality of producer threads isreferred to herein as the producer side of a data mover. The pluralityof consumer threads is referred to as the consumer side.

Informed by performance monitoring, a data mover can determine: (a) Ifmore threads are possible on either the producer or consumer sides; (b)whether increasing the number of threads on either side improves datamover throughput; and (c) whether more or fewer threads are needed, onone side or the other, to achieve a relative throughput balance betweenthe producer and consumer sides.

Achieving relative throughput balance in a data mover between theproducer and consumer sides is a desired aspect of embodiments inaccordance with the present invention. If a data mover is taking in data(producer side) significantly faster than it can discharge it (consumerside), a bottleneck exists wherein data accumulates at the data mover,for example in a buffer, and fails to move on toward the backup serverin a timely fashion. If this imbalance persists long enough, the buffercan fill up, and the data mover can only accept additional data asbuffer space is released and made available. Alternatively, if theconsumer side is significantly faster than the producer side, the datamover may have unused capacity that could be put into service byincreasing throughput on the producer side. Thus, if there exists asignificant imbalance between the producer and consumer sides it islikely that the data mover is failing to deliver its maximum throughput.

In embodiments in according with the present invention, a data mover canmonitor the balance between its producer and consumer sides and can addor remove threads on either side, as appropriate, so as to approach abalanced condition. Further detail is given with reference to FIG. 3Cand the detailed description thereof.

Advantages afforded by embodiments in accordance with the presentinvention include: (a) dynamic and continual modification of workloaddistribution among data movers, informed by performance monitoring andpeer-to-peer communication among data movers, during a backup cycle; (b)improved allocation of workload on subsequent backup cycles, can bebased at least in part on performance results of previous cycles; and(c) dynamic modification of internal parallelism in a data mover, duringa backup cycle, to achieve optimal throughput of the data mover.

Example embodiments in accordance with the present invention will now bedescribed in detail with reference to the drawing figures. FIG. 1 is ageneric representation of a computing environment, generally designatedwith numeral 100, within which embodiments in accordance with thepresent invention may operate.

Virtual machine host 110 comprises one or more physical systems hostingone or more virtual machines, designated VM 112.

Backup solution 105 comprises one or more of data mover 115, andsoftware (not shown) which operates during, and between, backup cycles.Detailed discussion of the software is found below with reference toFIGS. 2, 3A, 3B, 3C and the detailed descriptions thereof.

Data mover 115 reads data from one or more instances of VM 112 and sendsthe data to backup server 125. Data movers 115 communicate among eachother, and with backup solution 105, such communication represented aspeer to peer communication 116 or peer to server communication (notshown).

VM 112 and data mover 115 may be hosted on the same or on differentphysical or virtual machines. Data being backed up is designated asworkload 113, which is transported through data mover 115 andsubsequently sent to backup server 125, via network 120. Backup server125 may comprise one or more physical servers. Backup server 125 can beany machine or combination of machines capable of receiving, storing,retrieving and sending data, such as for example, a cloud-based objectgrid data storage solution. Workload 113 comprises the totality of datato be backed up by backup solution 105.

The terms “workload subset” and “subset” are synonymously defined hereinas that portion of workload 113 assigned to a data mover 115, theportion of workload originating from a single instance of VM 112. Theportion of workload 113 that is assigned to a data mover 115 can thus beconsidered as a series of one or more subsets of workload 113, eachsubset originating from a distinct instance of VM 112.

During a backup cycle, in response to certain circumstances, a subset ofworkload 113 can be reassigned among the plurality of instances of datamover 115 within backup solution 105, the purpose of the reassignmentgenerally being to improve overall performance of backup solution 105.However, reassignments could be made for reasons other than improvingoverall performance, such as for example, to reassign workload from adata mover 115 that fails, loses its network connectivity, or isotherwise unable to process some or all of its assigned workload 113.Further details, relative to reassignment, particularly for throughputimprovement, can be found below, with reference to FIG. 3C and thedetailed description thereof.

In general, network 120 can be any combination of connections andprotocols that will support communications between data mover 115 andbackup server 125. Network 120 can include, for example, a local areanetwork (LAN), a wide area network (WAN) such as the internet, or anycombination of the preceding, and can further include wired, wireless,and/or fiber optic connections.

FIG. 2 represents a data mover 115, generally designated with numeral200, in an embodiment in accordance with the present invention. Datamover 115 comprises buffer 220 into which incoming workload 113 iswritten by at least one producer thread 215. Outgoing workload 113 isread from buffer 220 by at least one consumer thread 225, the outgoingworkload 113 being transmitted to backup server 125 via network 120.Data mover 115 is operationally coupled with low level thread 210, highlevel thread 205, and performance model 235.

As used herein, the terms “local” and “peer” refer to the data movers. Adata mover referenced in particular, is referred to as a “local datamover”. From the perspective of a local data mover, other data movers inbackup solution 105 are “peer data movers”. The same distinction(local/peer) will be used, when convenient, with reference to workload,performance model, low and high level threads, and other aspects oflocal and peer data movers, wherein for example, local workload refersto the workload being processed by a local data mover, whereas peerworkload refers to workload being processed by one or more peer datamovers.

In at least one embodiment in accordance with the present invention,backup solution 105 includes: a) top level functions; b) high levelthreads 205; and c) low level thread 210. Top level functionsinclude: 1) overall coordination of backup solution 105; 2)identification of available data movers and the distribution of workload113 among the data movers; and 3) continuity of data mover performancestatistics from one backup cycle to the next.

FIG. 3A is a flowchart depicting the top level functions of backupsolution 105 operating in an embodiment in accordance with the presentinvention, and generally designated with numeral 300A. Among the toplevel functions of backup solution 105 are: 1) to assign instances ofdata mover 115 to instances of VM 112; and 2) to start instances of datamover 115 and launch a high level thread 205 and a low level thread 210associated with each data mover 115.

Referring to FIG. 3A, in response to initiating a backup cycle (functionblock 301), a set of VM 112 machines, which are to be backed up, isdetermined (function block 304) with reference to an inventory of VM 112machines (database 302). A set of available instances of data mover 115nodes is identified (function block 308) with reference to at least aninventory of instances of data mover 115 and their respectiveconnectivities (database 306). If a backup cycle is being performed forthe first time (for a given installation of backup solution 105)(decision 314, “Yes” branch), workload 113 mapping is determined inaccordance with algorithms pre-defined as aspects of an embodiment inaccordance with the present invention (function block 312).

If a backup cycle is being run subsequent to a first time (decision 314,“No” branch), instances of VM 112 which were not included in theprevious backup cycle, if any, are assigned to instances of data mover115 (function block 316). With reference to at least normalizedperformance ratios determined in one or more previous backup cycles(database 324), existing workload 113 mapping can be modified (functionblock 316).

One or more data movers are started (function block 318) and may operateconcurrently. High level threads 205 and low level threads 210 arestarted (function blocks 320 and 322 respectively), there being a highlevel thread 205 and a low level thread 210 in operational concert witheach data mover 115. It should be understood that there will exist aone-to-one correspondence between high level threads 205 and instancesof data mover 115 and between low level threads 210 and instances ofdata mover 115.

FIG. 3B is a flowchart, generally designated with numeral 300B,illustrating functions of high level thread 205, performed by a localdata mover 115 in at least one embodiment in accordance with the presentinvention. FIG. 3B continues from FIG. 3A.

A local instance of high level thread 205 is operationally coupled withlocal data mover 115. High level thread 205 controls at least theinstrumented monitoring of a local data mover 115, computation ofperformance statistics based at least in part on results of theinstrumented monitoring, communications with instances of peer datamover 115 and/or backup solution 105, and shifting of workload 113between a local data mover 115 and instances of peer data mover 115.

High level thread 205, when started, may initially enter a wait state(decision 332 and its “No” branch, respectively). Responsive tocompletion of a local subset, by a local data mover 115 (decision 332,“Yes” branch), performance statistics for local data mover 115, thestatistics associated with the subset just completed, are computed(function block 334). Performance model 235 is updated (function block336), and replicated at peer data movers 112 (function block 338).

Performance statistics of peer data movers 115 are similarly replicatedat local data mover 115 as processing of peer workload subsets iscompleted. Performance statistics of local and peer data movers 112 areanalyzed to determine what changes, if any, in the distribution ofworkload 113 among instances of data mover 115 appear likely to improvethroughput of backup solution 105 (function block 340). The analysis mayinclude determining performance ratios for a local data mover 115,relative to each VM 112 class processed by the local data mover 115, andcomparing the local performance ratios with peer performance ratios.Based at least in part on the analysis, remaining workload may beshifted between local data mover 115 and instances of peer data mover115 (function block 340).

If local workload 113 has not been completed (function block 330, “No”branch), high level thread resumes a wait state (function block 332 andits “No” branch).

If local workload 113 has been completed (function block 330, “Yes”branch), but peer workload 113 is still pending at instances of peerdata mover 115 (function block 342, “No” branch), local data mover 115may acquire workload from at least one peer data mover 115 (functionblock 344), the newly acquired workload becoming local workload 113.Responsive to acquiring additional local workload 113, high level thread205 enters a wait state (decision 332 and its “No” branch,respectively). If local and peer workloads are complete (function block342, “Yes” branch), high level thread may perform end-of-cycle tasks(function block 346) such as sending final performance results,including performance ratios, to the top level thread of FIG. 3A, afterwhich processing of high level thread 205 ends.

FIG. 3C is a flowchart illustrating functions of a low level thread 210,generally designated with numeral 300C, performed by data mover 115,operating in at least one embodiment in accordance with the presentinvention. FIG. 3C continues from FIG. 3A.

An instance of low level thread 210 is operationally coupled with datamover 115 to control one or more producer threads 215 and one or moreconsumer threads 225. Low level thread 210 optimizes performance oflocal data mover 115 by increasing or decreasing the numbers of producerthreads 215 and consumer threads 225, to achieve a maximum practicalthroughput of data mover 115. The number of producer threads 215operating concurrently in data mover 115, and the number of consumerthreads 225 operating concurrently in data mover 115 need not be equal.

If processing of local workload 113 has been completed (decision 350,“Yes” branch), processing ends. If local workload 113 has not beencompleted (decision 350, “No” branch) and local data mover 115 is ableto start backing up a new subset (decision 352, “Yes” branch), datamover 115 starts as many new subsets as it is able to start (functionblock 354 and decision 352, “Yes” branch, respectively and iteratively)from its assigned workload.

If data mover 115 is not able to start a new virtual machine backup(decision 352, “No” branch), data mover 115 optimizes the numbers of“producer” and “consumer” threads operating. If data mover 115 is ableto add a producer thread 215 (decision 356, “Yes” branch), it adds theproducer thread 215 (function block 358). If throughput of data mover115 does not improve in response to adding the producer thread 215(decision 360, “No” branch), data mover 115 removes a producer thread215 (function block 362).

If data mover 115 can add a consumer thread 225 (decision 364, “Yes”branch), it adds a consumer thread 225 (function block 366). Ifthroughput of data mover 115 does not improve in response to adding theconsumer thread 225 (decision 368, “No” branch), data mover 115 removesa consumer thread 225 (function block 370).

Responsive to any of the following enumerated conditions or events (athrough e), data mover 115 performs a comparison (decision 372) todetermine the relative throughput balance between its “producer” and“consumer” sides: a) Data mover 115 cannot add a consumer thread 225(decision 364, “No” branch); b) Data mover 115 adds a consumer thread225 and throughput improves (decision 368, “Yes” branch); c) Data mover115 removes a consumer thread 225 (function block 370); d) Data mover115 removes a consumer thread 225 (function block 376); and e) Datamover 115 removes a producer thread 215 (function block 374).

If throughput of data mover 115's consumer side exceeds, by an amountgreater than a specified threshold value, the throughput of its producerside (decision 372, “C-Fast” branch), a consumer thread 225 is removed(function block 376) and the comparison is repeated (decision 372, “CFast” branch and block 376) iteratively, until the consumer sidethroughput no longer exceeds the producer side throughput by more thanthe specified threshold.

If throughput of data mover 115's producer side exceeds, by an amountgreater than a specified threshold value, the throughput of its consumerside (decision 372, “P-Fast” branch), a producer thread 215 is removed(function block 374) and the comparison is repeated (decision 372, “PFast” branch and block 374) iteratively, until the producer sidethroughput no longer exceeds the consumer side throughput by more thanthe specified threshold.

If the difference between throughputs of the producer side and theconsumer side is less than a specified threshold value, data mover 115is considered to be in balance (decision 372, “Balanced” branch), inresponse to which data mover 115 determines whether local workload 113has been completed (decision 350). If local workload 113 has beencompleted (decision 350, “Yes” branch), processing ends.

When a data mover 115, within backup solution 105 is hosted on a virtualmachine that is also served by backup solution 105, the data mover is,by default, assigned workload from the virtual machine on which the datamover resides. Such a case is defined herein as a “trivial” case.Trivial cases are generally excluded from workload shifting andrandomization aspects of embodiments in accordance with the presentinvention, based on the assumption that a data mover hosted on a virtualmachine that is being backed up, will likely have the greatestperformance due to the close network proximity the data mover and thevirtual machine have to each other. However, it is to be understood thatthis disclosure does not preclude trivial cases from inclusion inworkload shifting and randomization aspects, nor does it preclude a datamover operating in a trivial case from being assigned workload fromother virtual machines in addition to the trivial virtual machine.

One aspect of embodiments in accordance with the present invention is agoal to distribute workload from as many non-trivial VM 112 classes asis practical across as many instances of data mover 115 as practical,such that backup solution 105 converges toward an optimal configurationwith respect to data mover/virtual machine class combinations. Tofurther this goal, “randomization” is conducted periodically betweenbackup cycles so that a data mover 115's performance, when processing asmany distinct virtual machine classes as is practical, can be measured.This randomization could be conducted according to the following rules,or to other rules as established in various implementations:

A) If a virtual machine class exists which has not yet been processed bya particular data mover 115, select at least one instance of VM 112 fromthe virtual machine class and process workload from the selected atleast one virtual machine instances through the data mover. The basisused to select a VM 112 could be a random or pseudo-random selectionalgorithm or other selection method.

B) Periodically, some instances of VM 112, of a particular class, can beassigned to a particular data mover 115, if the data mover hasn'trecently processed workload from that virtual machine class.

To modify mapping of workload 113 in embodiments in accordance with thepresent invention, local data mover 115 communicates with instances ofpeer data mover 115 to update its local internal model and the localmodels of instances of peer data mover 115. Local data mover 115 cananalyze its own performance ratios by VM 112 class(es), and compare itsratios to those of instances of peer data mover 115 operating on thesame VM 112 class(es). Local data mover 115 can offer, to instances ofpeer data movers 115, one or more subsets of workload 113 from virtualmachines of a class which local data mover 115 achieves lowerthroughput. Local data mover 115 can also seek to acquire one or moreworkload subsets for which it achieves higher throughput, the one ormore workload subsets offered by one or more instances of peer datamover 115. In some embodiments in accordance with the present invention,only a few workload subsets are moved at any one time, so as to avoidsudden performance fluctuations of backup solution 105.

In the foregoing discussion relative to workload shifting andrandomization, the quantitative definitions of the phrases “best,”“periodically,” “some,” “recently,” and “few” are implementation aspectsin embodiments in accordance with the present invention.

After completion of a backup cycle, at least the final mapping ofinstances of data mover 115 to instances of VM 112, and the respectiveperformance statistics, are retained for use in the next back up cycle,to maintain continuity of the optimization that has taken place, and toprovide a basis from which to establish initial operating parameters forthe next backup cycle.

FIG. 4 depicts a block diagram of components of data processing system400, representative of any computing system within data processingenvironment 100 in accordance with an illustrative embodiment of thepresent invention. It should be appreciated that FIG. 4 provides only anillustration of one implementation and does not imply any limitationswith regard to the environments in which different embodiments may beimplemented. Many modifications to the depicted environment may be made.

Data processing system 400 includes communications fabric 402, whichprovides communications between computer processor(s) 404, memory 406,persistent storage 408, communications unit 410, and input/output (I/O)interface(s) 412. Communications fabric 402 can be implemented with anyarchitecture designed for passing data and/or control informationbetween processors (such as microprocessors, communications and networkprocessors, etc.), system memory, peripheral devices, and any otherhardware components within a system. For example, communications fabric402 can be implemented with one or more buses.

Memory 406 and persistent storage 408 are computer-readable storagemedia. In this embodiment, memory 406 includes random access memory(RAM) 414 and cache memory, which may include cache 416. In general,memory 406 can include any suitable volatile or non-volatilecomputer-readable storage media. Memory 406 and persistent storage 408may be logically partitioned and allocated to one or more virtualmachines and/or virtual volumes.

Computer programs and processes are stored in persistent storage 408 forexecution by one or more of the respective computer processors 404 viaone or more memories of memory 406. For example, processes implementingand managing thinly provisioned volumes may be stored in persistentstorage 408. In this embodiment, persistent storage 408 includes amagnetic hard disk drive. Alternatively, or in addition to a magnetichard disk drive, persistent storage 408 can include a solid state harddrive, a semiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer-readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 408 may also be removable. Forexample, a removable hard drive may be used for persistent storage 408.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer-readable storage medium that is also part of persistent storage408.

Communications unit 410, in these examples, provides for communicationswith other data processing systems or devices, including other computingsystems of storage system 102. In these examples, communications unit410 includes one or more network interface cards. Communications unit410 may provide communications through the use of either or bothphysical and wireless communications links. Computer programs andprocesses may be downloaded to persistent storage 408 throughcommunications unit 410.

I/O interface(s) 412 allows for input and output of data with otherdevices that may be connected to data processing system 400. Forexample, I/O interface 412 may provide a connection to external devices418 such as a keyboard, keypad, a touch screen, and/or some othersuitable input device. External devices 418 can also include portablecomputer-readable storage media such as, for example, thumb drives,portable optical or magnetic disks, and memory cards. Software and dataused to practice embodiments of the present invention can be stored onsuch portable computer-readable storage media and can be loaded ontopersistent storage 408 via I/O interface(s) 412. I/O interface(s) 412may also connect to a display 420.

Display 420 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The terminology used herein was chosen to best explain the principles ofthe embodiment, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

What is claimed is:
 1. A method for optimizing performance of a computerbackup solution that includes at least a first data mover and a seconddata mover, the method comprising: measuring performance statistics ofthe first data mover with respect to at least one virtual machine classprocessed by the first data mover, to produce performance statistics, byvirtual machine class, of the first data mover; measuring performancestatistics of the second data mover with respect to at least one virtualmachine class processed by the second data mover, to produce performancestatistics, by virtual machine class, of the second data mover;optimizing, by one or more processors, performance of the data mover;operating, by one or more processors, a performance model operationallycoupled with a data mover wherein the performance model includes atleast performance statistics of the first data mover and performancestatistics of the second data mover; initiating, by one or moreprocessors, communications between the first data mover and the seconddata mover during at least a backup cycle, to exchange at leastperformance statistics between the first data mover and the second datamover; generating, by one or more processors, a peer-to-peer modelwherein the first data mover has access to the performance model of thesecond data mover and the second data mover has access to theperformance model of the first data mover; computing, by one or moreprocessors, performance rankings with respect to virtual machine class,of the first data mover and the second data mover, based at least inpart on performance statistics of the first data mover and performancestatistics of the second data mover; analyzing, by one or moreprocessors, respective performance statistics of the first data moverand the second data mover, with respect to virtual machine class, toproduce performance rankings, by virtual machine class, of the firstdata mover and the second data mover; shifting, by one or moreprocessors, some workload from the first data mover to the second datamover, in accordance with their respective performance rankings, withrespect to virtual machine class, such that the computer backup solutionconverges toward an optimized configuration; shifting, by one or moreprocessors, some workload from virtual machines of a class, from thefirst data mover to the second data mover, based at least in part, onthe performance rankings of the first data mover and the second datamover with respect to workload from virtual machines of the class;operating concurrently, one or more producer threads in the data mover;operating concurrently, one or more consumer threads in the data mover;changing the number of producer threads or consumer threads operatingconcurrently in the data mover; measuring a change of performance of thedata mover, in response to changing the number of producer threads orconsumer threads operating concurrently in the data mover, to produceperformance statistics on which to base, at least in part, a subsequentchange in the number of producer threads or consumer threads operatingconcurrently in the data mover, such that the data mover convergestoward an optimized performance state; storing at least performancestatistics relative to the first data mover, in the performance modelassociated with the first data mover; storing at least performancestatistics relative to the first data mover, in the performance modelassociated with the second data mover; storing at least performancestatistics relative to the second data mover, in the performance modelassociated with the first data mover; and storing at least performancestatistics relative to the second data mover, in the performance modelassociated with the second data mover.